Robust anchoring of annotations to content

ABSTRACT

Annotations are robustly anchored to content. The robust anchoring allows annotations to be associated with the content independently of the format used to store the content. The annotations can be re-anchored to the content after the content has been modified, including modifications to the format used to store the content.

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent file or records, but otherwise reserves all copyrightrights whatsoever.

TECHNICAL FIELD

This invention relates to annotating content, and more particularly torobust anchoring of annotations to content.

BACKGROUND

As computing technology has advanced, increasingly powerful computingdevices have become available. Such devices have led to an increase inthe number of computing devices being used as well as an expansion ofthe manner in which these devices are being used. One such usage forcomputing devices is the generation and distribution of electronic ordigital documents.

The transition to content authoring, revising, and distributing usingcomputers rather than pen (or typewriter) and paper has created manybenefits, but has not been without its problems. Some of these problemsare rooted in the different manner in which content is authored andrevised on computers rather than the more traditional paper methods. Onespecific problem is the annotating of content. Paper documents have beentraditionally annotated manually, such as by highlighting or underliningof text and/or notes written in the margin of the paper. Annotatingdigital documents (e.g., documents which are being authored and revisedon a computer) in a similar manner is difficult because digitaldocuments are easily and frequently altered, thereby changing theportion of the document to which the annotation corresponds. It wouldthus be beneficial to provide a way to improve the manner in whichannotations are associated with portions of documents so that theannotations are still associated with the correct portion of thedocument despite alterations to the underlying content.

SUMMARY

Robust anchoring of annotations to content is described herein.

According to one aspect, an annotation is associated with a particularportion of content by capturing multiple features of the portion. Thesefeatures include one or more features describing a beginning point ofthe portion, one or more features describing an ending point of theportion, and one or more features describing the portion between thebeginning point and the ending point. According to one implementation,these one or more features are captured independently of a format usedto store the content.

According to another aspect, an annotation anchor that describes aregion of content to which an annotation corresponds is reassociatedwith the content (or a modified version thereof). One or more features(for example, keywords) in the annotation anchor are identified, and oneor more candidate regions of the modified version to which theannotation potentially corresponds are identified. A score is generatedfor the one or more regions, and the identified region having the bestscore is selected as the region of the modified version of the originalcontent to which the annotation corresponds.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an exemplary system for anchoringan annotation to content.

FIG. 2 illustrates exemplary content with a highlighted portioncorresponding to an annotation.

FIG. 3 illustrates an exemplary process carried out to generate a set ofone or more keywords that exist within a region.

FIG. 4 is a flowchart illustrating an exemplary process for generatingan annotation anchor.

FIG. 5 is a flowchart illustrating an exemplary process for re-anchoringan annotation to modified content.

FIGS. 6 a, 6 b, and 6 c are flowcharts illustrating an exemplary processfor generating a candidate annotation region and score corresponding toa selected keyword.

FIGS. 7, 8, 9, 10, 11, and 12 illustrate exemplary user interfaces thatmay be presented by a UI module.

FIG. 13 illustrates a more general exemplary computer environment, whichcan be used to implement the improved meta data management describedherein.

DETAILED DESCRIPTION

Robust anchoring of annotations to content is described herein. For eachannotation, information describing the portion of the content to whichthe annotation corresponds is robustly captured, thereby “anchoring” theannotation to the portion of the content. This captured information,also referred to as an “anchor”, includes information regarding abeginning point of the portion, an ending point of the portion, and theregion between the two points. Subsequently, the content (includingpossibly the portion to which the annotation corresponds) can bemodified, and the annotation re-anchored to the modified content.

FIG. 1 is a block diagram illustrating an exemplary system 100 foranchoring an annotation to content. System 100 includes an annotationmarking engine 102 and an annotation re-anchoring engine 104. Engines102 and 104 may be implemented in the same computer, or alternatively indifferent computers. During operation, an indication 106 of anannotation corresponding to underlying content as well as the underlyingcontent 108 are received at annotation marking engine 102. Indication106 includes an identification of the portion of content 108 to whichthe annotation corresponds. This identification can be made in any of awide variety of conventional manners, such as by identifying the pagenumber, line number, and character number of where the portion beginsand ends. Indication 106 can be received in any of a variety of manners.For example, a content authoring module (not shown) may receive an inputfrom a user highlighting a portion of the content, and an indication ofthis input forwarded to engine 102. Annotations can correspond to aportion of content 108 explicitly identified by a user, or alternativelya single point explicitly identified by the user (and the portion ofcontent 108 being implicitly identified).

Underlying content 108 can be any of one or more types of content, suchas text content, audio content, video content, image content, etc., orcombinations of one or more of these types of content. Underlyingcontent 108 can be any type of content from which one or more partially-or uniquely-identifying robust features can be extracted. Similarly, theannotation generated by a user can be in any of a variety of types ofcontent (e.g., text content, audio content, video content, imagecontent, etc., or combinations thereof), and need not be of the sametype of content as the underlying content 108 that it annotates. Any ofa wide variety of conventional components can be used to generate theunderlying content and/or the annotation content, including text editorsand keyboards, microphones, image capture devices, etc.

Upon receipt of indication 106, annotation marking engine 102 robustlycaptures the beginning point of the portion, ending point of theportion, and region between the beginning and ending points of theregion. This capturing is performed by a beginning point capture module110, an ending point capture module 112, and a region capture module114, respectively.

FIG. 2 illustrates exemplary content with a highlighted portion orregion corresponding to an annotation. In FIG. 2, an exemplary sentenceis illustrated on a display 120 with a portion 122 highlighted. Althougha dashed box is used on display 120 to indicate highlighting,alternatively highlighting can be indicated in a variety of othermanners (such as changing the color or font of the highlighted text,marking the area surrounding the text with a different color, animatingthe text, etc.). With respect to highlighted portion 122, the beginningpoint 124 of the portion is at the letter “h” in “hardwired”, the endingpoint 126 of the portion is at the letter “n” in “brain”, and the region128 is the characters between these points, inclusive (that is,“hardwired into your brain”).

For text content, the beginning point refers to the first character inthe highlighted portion, while the ending point refers to the lastcharacter in the highlighted portion. Alternatively, these points may beimplemented in different manners. For example, the beginning point maybe the character immediately preceding the first highlighted character,or the area between the first highlighted character and the characterimmediately preceding the first highlighted character. Similarly, theending point may be the character immediately succeeding the lasthighlighted character, or the area between the last highlightedcharacter and the character immediately succeeding the last highlightedcharacter.

Returning to FIG. 1, beginning point capture module 110 extracts one ormore features regarding the beginning of the portion to which theannotation corresponds. Any of a wide variety of features regarding thebeginning point of the portion can be extracted. One example of such afeature is the actual characters of the content at the beginning point(e.g., one or more characters immediately preceding and/or succeedingthe beginning point). In one implementation, the fifteen charactersbefore the beginning point and the fifteen characters after thebeginning point are captured as the features of the beginning point.Another example of such a feature is a hash value of the characters atthe beginning point (e.g., a hash value generated by using aconventional hashing function (such as MD5 (Message Digest 5), SH-1(Secure Hash Algorithm-1), or other cryptographic or non-cryptographichash functions) to hash one or more characters immediately preceding orsucceeding the beginning point). Yet another example of such a featureis an offset of the beginning point relative to another point (e.g.,relative to the beginning or ending of the entire document, thebeginning or ending of a particular section of the document, etc.). Suchan offset can be calculated in different manners, such as a number ofcharacters, a number of words, a number of paragraphs, a number ofpixels, a particular length (e.g., number of inches) on a printed copyof the document, and so on.

Another example of such a feature is punctuation at or near (e.g.,within a particular number of characters of) the beginning point. Stillanother example of such a feature is one or more keywords or propernouns at or near (e.g., within a particular number of characters of) thebeginning point. Yet another example of such a feature is dependent onthe implemented document structure (e.g., the position of the beginningpoint in a HyperText Markup Language (HTML), Portable Document Format(PDF), or Rich Text Format (RTF) parse tree).

Ending point capture module 112 extracts one or more features regardingthe ending of the portion to which the annotation corresponds. Analogousto beginning point capture module 110, any of a wide variety of featuresregarding the ending point of the portion can be extracted. In oneimplementation, the fifteen characters before the ending point and thefifteen characters after the ending point are captured as the featuresof the ending point. Ending point capture module 112 may extract thesame types of features as beginning point capture module 110 (e.g., bothmay capture actual characters), or different types of features (e.g.,module 110 may capture actual characters and an offset, while module 112may capture a hash value of the characters and an offset).

Region capture module 114 extracts one or more features regarding theportion to which the annotation corresponds (that is, the region betweenthe beginning and ending points). Various different features can beextracted by module 114. One such feature is the length of the portion.This length can be expressed in different manners, such as a characteror word count, a pixel count, a length of the portion (e.g., in inches)when printed, etc.

Another feature that can be extracted by module 114 is a set of one ormore keywords that exist within the region. FIG. 3 illustrates anexemplary process 140 carried out by module 114 to generate a set of oneor more keywords that exist within the region. Process 140 mayoptionally be implemented in software, firmware, hardware, or acombination thereof. Initially, a histogram that identifies wordfrequencies in the document is generated (act 142). It is to beappreciated that the histogram is generated in act 142 need only begenerated once for each document (and not every time an anchor iscreated). The histogram generated in act 142 identifies, for each wordin the document, the frequency of that word (i.e., the number of timesthe word appears in the document). This generation in act 142 is overthe entire document, not just the highlighted region. The words thatappear in the highlighted region are then identified (act 144). Thefrequency of the words in the highlighted region are then identifiedfrom the histogram (act 146). The frequency identified for each word inact 146 is the frequency of that word across the whole document, notjust the frequency within the highlighted region.

Once the frequencies are identified, one or more words having the lowestfrequencies are selected from the highlighted region (act 148). Thenumber of words selected can vary, and in one implementation isdependent on the length of the region (e.g., the number of selectedwords may be a particular percentage of the number of words in theregion). In one implementation, at least three keywords are selected.Alternatively, a particular number of words may not be selected, ratherthe words selected are those one or more words that have the lowestfrequency (and all of the words with the lowest frequency are selected,regardless of how many there are). For example, one region of fifteenwords may have three words that all have the lowest frequency (e.g.,appearing five times each in the document while all others appear six ormore times each), while another region of fifteen words may have onlyone word with the lowest frequency (e.g., appearing three times in thedocument while all others appear four or more times each). In the firstinstance, all three words would be selected as keywords, while in thesecond instance only the one word would be selected as a keyword.

Returning to FIG. 1, additional features that may be extracted from thehighlighted region are the distance between the beginning point of theregion and each keyword, as well as the distance between each keywordand the ending point of the region. These distances can be measured in avariety of different manners, such as characters, words, pixel counts,inches (e.g., on the document when printed), etc. Given these distancesfrom each keyword to beginning and ending points, the distances betweenkeywords can be readily generated and used during re-anchoring, asdiscussed in more detail below.

Various other features may also be extracted for the highlighted region,such as punctuation marks within the region (e.g., whether the beginningor ending points are located at particular punctuation marks, such ascommas, periods, or quotes), sentence boundaries within the region(e.g., whether the beginning point is the beginning of a sentence, orwhether the ending point is the ending of a sentence), proper nounswithin the region, the grammatical structure of the region (e.g.,subject/object position information, types of clauses within the region,etc.), an implementation-dependent document structure (e.g., an HTML,PDF, or RTF parse tree of the region), “fingerprinting” of the document(e.g., generate hashes of the entire document in short segments, thenattach annotations to this “hash topography” of the document), thesemantic “meaning” of the highlighted region (e.g., using NaturalLanguage Processing techniques to analyze and store information aboutthe “meaning” of what was selected in the region, and so forth.

Once annotation marking engine 102 has captured the various features forthe beginning point, ending point, and region in-between these points,the captured features are output as an annotation anchor 152. Theannotation anchor thus describes various features or aspects of theportion of content 108 to which the annotation corresponds. Given theinformation in the annotation anchor 152, the anchor 152 can be easilystored separately from the underlying content. Alternatively, ifdesired, the anchor 152 could be stored with the underlying content 108or with the annotation content.

The following data structures illustrate one exemplary way in whichcaptured data for annotation anchors can be stored. It is to beappreciated that these are exemplary only, and that alternativestructures may also be used. typedef struct _sRAnnPosInfo { sRPointInfofront; //beginning point sRPointInfo back; //ending point KEYWORDLISTkeywords; //list of _sRKeywordInfo structures int length; //length ofthe annotation region _bstr_t bkmark; //alternate bookmark for fasterre-anchoring //if underlying content did not change long offset;//offset from the beginning of document to beginning //of annotationregion } sRAnnPosInfo; typedef struct _sRPointInfo { _bstr_tleftContent; //the 15 characters before the point _bstr_t rightContent;//the 15 characters after the point } sRPointInfo; typedef struct_sRKeywordInfo{ _bstr_t keyword; //the keyword int distToStart;//distance from start of keyword to start of annotation int distToEnd;//distance from start of keyword to end of annotation } sRKeywordInfo;

FIG. 4 is a flowchart illustrating an exemplary process for generatingan annotation anchor. The process 160 of FIG. 4 is carried out byannotation marking engine 102 of FIG. 1, and may be implemented insoftware, firmware, hardware, or a combination thereof.

Initially, features corresponding to the beginning point of theannotation are captured (act 162), and features corresponding to theending point of the annotation are captured (act 164). Featurescorresponding to the region between the beginning point and ending pointare also captured (act 166). These captured features are then output asthe annotation anchor for the annotation (act 168).

Returning to FIG. 1, content 108 can also be input to a content editor154. Content editor 154 represents any device or module capable ofaltering content 108. For example, content editor 154 may be a wordprocessing application. The content 108 may be altered by editor 154prior to, subsequent to, or concurrently with the marking of annotationsperformed by engine 102. The altered or modified content 156 is outputby editor 154 and made available to annotation re-anchoring engine 104.Additionally, the annotation anchors 152 from annotation marking engine102 are also made available to annotation re-anchoring engine 104.Re-anchoring engine 104 uses annotation anchors 152, as well as modifiedcontent 156, to re-anchor the annotations to modified content 156,outputting re-anchored annotations 158. Although discussed hereinprimarily with reference to re-anchoring annotations to modifiedcontent, it should be noted that annotations can be re-anchored tounmodified content as well (or re-anchored to content that has beenstored in a different format).

An example of the altering of the underlying content as well as there-anchoring of annotations to the altered content is illustrated inFIG. 2. The display 180 illustrates a sentence that is a modifiedversion of the sentence illustrated in display 120. The originalannotation is anchored to the highlighted portion 122 in display 120.However, since the sentence has been altered, the exact highlightedportion 122 (the phrase “hardwired into your brain”) does not exist ondisplay 180. Thus, annotation re-anchoring engine 104 of FIG. 1re-anchors the annotation to the highlighted portion 182. Thus, eventhough the wording of the underlying content has been changed, theannotation is still anchored to the portion of the content that reflects(or closely reflects) the same idea as was originally annotated giventhe content in display 120.

Annotation re-anchoring engine 104 includes a candidate regiondetermination module 186, a score comparator module 188, a userinterface (UI) module 190, and a preferences module 192. For eachannotation to content 108, candidate region determination module 186attempts to generate a score (based on the annotation anchor for thatannotation) for one or more regions of modified content 156 to which theannotation may potentially correspond. The score for a candidate regionreflects how well the candidate region matches the region in theoriginal content to which the annotation corresponds. Score comparatormodule 188 analyzes the various scores generated by candidate regiondetermination module 186 and attempts to identify one of those candidateregions, based on the scores, to anchor the annotation to. Depending onthe various scores, situations may arise where the user is prompted forinput regarding where a particular annotation should be anchored. Inthese situations, UI module 190 allows the user to enter such input.Additionally, the behavior of one or more of modules 186, 188, and 190may be user-configurable, in which case the user-configured options arestored as preferences 192. The behavior of annotation re-anchoringengine 104 is described in additional detail with reference to thefollowing figures.

FIG. 5 is a flowchart illustrating an exemplary process for re-anchoringan annotation to modified content. The process of FIG. 5 is carried outby annotation re-anchoring engine 104 of FIG. 1 and may be implementedin software, firmware, hardware, or a combination thereof.

Initially, an annotation anchor is received (act 222). The keywords fromthe annotation anchor are identified (act 224), and one of the keywordsis selected (act 226). Any one of the keywords can be selected in act226 (e.g., the first one from a list stored in the annotation anchor,one selected at random, etc.). A candidate annotation region and scorecorresponding to the selected keyword are then generated based on thelocation of the keyword (act 228). If the selected keyword appearsmultiple times within the modified content, then a candidate annotationregion and score is generated in act 228 for each of these multipleoccurrences of the keyword. This generation in act 228 is performedbased on the relationship of the keyword to any other keywords in theannotation anchor as well as the beginning and ending point informationin the annotation anchor. An exemplary process for implementing act 228is discussed in more detail below with reference to FIGS. 6 a-6 c.

A check is then made as to whether the generated score exceeds athreshold value (act 230). If the score does exceed the threshold valuethen the candidate region is determined to be the region that theannotation is to be anchored to, and the annotation is attached to thatregion (act 232). Attaching the annotation to the region refers toreplacing the previous indication of the portion of the content that theannotation previously corresponded to with an indication of the newregion. Thus, when the modified content is subsequently displayed, thenew region is highlighted for the annotation. The specific manner inwhich the annotation is attached to the region will vary byimplementation, and will typically use the same format as was used toindicate the initial region to engine 102. The threshold value used inact 230 can vary, but should be designed to be high enough thatdetermination of the correct region can be confidently made withoutanalysis of any additional candidate regions. For example, the thresholdvalue may be 98 on a scale from 0 to 100.

If the score does not exceed the threshold value in act 230, then acheck is made as to whether there are any additional keywords that havenot yet been selected (act 234). If there are any such additionalkeywords than the process returns to act 226 to select one of theremaining keywords. However, if there are no additional keywords, thenthe process proceeds to identify the largest score of those generated inact 228 (act 236). A check is then made as to whether the largest scoreexceeds another threshold value (act 238), referred to as the “guessthreshold”. The guess threshold value is lower than the threshold valuein act 230, but should be designed such that there is a high level ofconfidence in the determination that the candidate region is the correctregion if the candidate region's score is above the guess threshold, andsuch that there is a low level of confidence in the determination thatthe candidate region is the correct region if the candidate region'sscore is below the guess threshold. For example, the threshold value inact 238 may be 70 or 80 on a scale from 0 to 100.

If the identified score exceeds the threshold value in act 238, then thecandidate region corresponding to the identified score is determined tobe the region that the annotation is to be anchored to, and theannotation is attached to that region (act 240). Additionally, theinformation in the annotation anchor is re-captured based on the newregion the annotation is attached to (act 242). This re-calculationcomprises an annotation marking engine (e.g., engine 102 of FIG. 1)re-capturing the various beginning point, ending point, and regionfeatures based on the new region that the annotation corresponds to. Thehistory of the annotation anchoring may also be saved with theannotation anchor or alternatively elsewhere, allowing subsequentreference back to the preceding one or more annotation anchors ifdesired. Although not illustrated in FIG. 5, the annotation anchor mayalso be re-captured after attaching the anchor to the region in act 232.

Returning to act 238, if the identified score does not exceed thethreshold value, then a check is made as to whether the identified scoreis less than a lower bound (act 244), referred to as the “orphaningthreshold”. This lower bound should be designed to be less than thethreshold value in act 238, and low enough that there is littleconfidence in the determination that the candidate region is the correctregion. For example, the lower bound in act 244 may be 30 or 40 on ascale from 0 to 100. If the identified score is below the lower bound,then the annotation is orphaned (act 246). Orphaning the annotationmeans that the annotation is not attached to any particular region ofthe content (all orphaned annotations may be displayed, for example, atthe end of the content). However, if the identified score is greaterthan the lower bound, then the user is queried regarding placement ofthe annotation (act 248). In this situation, the candidate region'sscore is greater than the orphaning threshold but less than the guessthreshold (there is sufficient information regarding the candidateregion to position the annotation in the document, but not enoughconfidence to be sure it belongs there). For example, UI module 190 ofFIG. 1 may display to the user an indication that the annotation isorphaned and also a “guess” button that the user can press to causeannotation to be anchored to the highest-scoring candidate region.Alternatively, UI module 190 may display to the user an identificationof the annotation and the candidate region, and allow the user to inputwhether the region is the correct region to which the annotation shouldbe attached. The user may thus identify which region is the correctregion for the annotation, or alternatively indicate that the annotationshould be orphaned.

The different threshold values discussed in acts 230 and 238, as well asthe lower bound discussed in act 244, may be user-configurableparameters (e.g., stored as preferences 192 of FIG. 1). When thesevalues are user-configurable parameters, UI module 190 presents aninterface to the user allowing the user to enter inputs as to what oneor more of these different values should be. The input by the user maybe specific values (e.g., the user may enter “80” to indicate that aparticular threshold amount should have a value of 80), or alternativelythe inputs may be implicit. For example, UI 190 may present to the useran interface querying the user as to whether the anchoring of aparticular annotation was correct. The results of one or more of suchqueries may then be used by engine 104 to learn more appropriatethreshold or lower bound values. One specific example is if engine 104automatically anchors multiple annotations because they have a scoreexceeding the threshold value (e.g., 80) in act 238, but the userindicates that many of those annotations are anchored incorrectly,engine 104 may increase the threshold value in act 238 to (e.g., to 84)in an attempt to decrease the number of annotations that are incorrectlyanchored.

Alternatively, acts 230 and 232 may optionally be bypassed in process220. Acts 230 and 232 operate as a shortcut to avoid processing numerouskeywords and regions of a document if a candidate region is found thatis almost certainly the right one (e.g., due to its very high score).Thus, acts 230 and 232 may be removed from process 220 (although, undercertain situations, this removal may result in increased processing timeto re-anchor an annotation).

Various other modifications may also be made to the process of FIG. 5.The process of FIG. 5 is a robust process that can correctly identifythe correct region to which an annotation should be anchored even thoughthe content of that region has been changed. While the process of FIG. 5also accurately identifies the correct region to which an annotationshould be anchored if the content of that region has not changed, othermethods may be able to anchor annotations to unchanged content morequickly. Thus, in an alternative embodiment, the annotation anchoringdescribed herein is modified to take advantage of the faster performanceof other algorithms to identify unchanged regions to which annotationsare to be anchored. In one implementation, a “getBookmark” method usedby the Microsoft Internet Explorer 5.58 web browser is used to identifyunchanged regions to which annotations are to be anchored. Additionalinformation regarding the “getBookmark” method is available fromMicrosoft Corporation of Redmond, Wash.

FIGS. 6 a-6 c are flowcharts illustrating an exemplary process 228 forgenerating a candidate annotation region and score corresponding to aselected keyword. The process 228 of FIGS. 6 a-6 c is carried out byannotation re-anchoring engine 104 of FIG. 1, and may be performed insoftware, firmware, hardware, or a combination thereof.

Initially, given the region information from the annotation anchor,multiple keywords in the modified content are identified (act 260).These keywords may be identified each time process 228 is performed, oralternatively only once per annotation anchor.

The score for the selected keyword is initialized to zero (act 262) anda check is made as to whether there are any additional keywords in theoriginal annotation region after the selected keyword (act 264). Thedistance information stored in the annotation anchor (e.g., distancesbetween beginning point and keywords and/or keywords and ending point)allows engine 104 to determine the order of keywords in the annotationregion. Thus, the next keyword in the original annotation region (ifany) can be readily identified. If there is at least one additionalkeyword after the selected keyword, a search is made through theidentified keywords in the modified content for the first occurrence ofthe keyword after the selected keyword (act 266). A check is then madeas to whether including the first occurrence of the next keyword in thecandidate region would keep the length of the candidate region less thantwice the length of the original region (act 268). If so, then 100points are added to the score (act 270), the candidate region isextended to include that keyword, and processing returns to act 264.However, if including the first occurrence of the next keyword in thecandidate region would not keep the length of the candidate region lessthan twice the length of the original region, then processing returns toact 264 without adding any value to the score and without extending thecandidate region to include that keyword.

The process continues to check whether there are any additional keywordsin the original annotation region after the selected keyword, and addspoints to the score for those keywords, until all of the additionalkeywords have been analyzed (acts 264-270). In other words, if theanchor information includes a list of n keywords, process 228 looks atall n-l other keywords for each of the keywords identified in act 260.This accounts for the possibility that the keywords get reordered in themodified content. For instance, suppose keywords A, B, and C areidentified in the anchor. The process looks for all occurrences of A inthe document (act 260). For each occurrence, the process looks foroccurrences of B and C that follow it in the content. If the processfails to find a candidate region with a high enough score, the processlooks for all occurrences of keyword B in the document. For eachoccurrence, the process looks for occurrences of A and C that follow itin the content, and so on.

After all of the additional keywords (if any) have been analyzed, theprocess looks for the first beginning point (e.g., point 184 of FIG. 2)preceding the selected keyword, based on the interior of the beginningpoint (act 272). The interior of the beginning point refers to the areabetween the beginning point and the selected keyword. The search in act272 begins at the selected keyword and works its way towards thebeginning of the content, analyzing each possible point along the way.The features from the annotation anchor are compared to each possiblepoint and a determination made as to whether a match exists. In oneimplementation, a “match” exists when the features from the annotationanchor are the same as (identical to) the features at the point beinganalyzed. For example, the 15 characters preceding the point beinganalyzed are the same as the 15 characters preceding the beginning point(and stored as a feature in the annotation anchor). Alternatively, a“match” may exist when there is only a partial match (e.g., a partialsubstring match). Any of a variety of conventional processes may be usedto determine a partial substring match, such as AGREP. (additionalinformation regarding AGREP is available from the University of Arizona,Department of Computer Science, Tucson, Ariz.).

Additionally, it should be noted that, analogous to the discussion aboveregarding extracting features for a “point”, the features thatdistinguish a “point” need not be literal character strings. Rather,such features could be a “fingerprint”; a weighted collection ofkeywords, proper nouns, and punctuation; a character offset; etc.

Processing then proceeds based on whether a beginning point is locatedin act 272 (act 274). If a beginning point is located, then a check ismade as to whether the distance from the located beginning point to theselected keyword exceeds the distance from the original beginning pointto the selected keyword (as indicated in the annotation anchor) bygreater than a threshold amount (e.g., greater than twice the originaldistance) (act 276). If the distance is not exceeded by greater than thethreshold amount, then 50 points are added to the score (act 278), thecandidate region is extended to include the located beginning point, andprocessing proceeds to look for the first ending point succeeding theselected keyword based on the interior of the selected keyword (act280). If the distance is exceeded by greater than the threshold amount,then processing proceeds to act 280 without adding any points to thescore and without extending the candidate region to include the locatedbeginning point.

At act 280, the first ending point (e.g., point 185 of FIG. 2) issearched for analogous to the search for the first beginning point inact 274, except that the search proceeds from the selected keywordtowards the end of the document. The interior of the ending point refersto the area between the ending point and the selected keyword. Analogousto the discussion above, a match may exist when the features at thepoint being analyzed are the same as or within a threshold amount of thefeatures in the annotation anchor.

Processing then proceeds based on whether an ending point is located inact 280 (act 282). If an ending point is located, then a check is madeas to whether the distance from the selected keyword to the locatedending point exceeds the distance from the selected keyword to theoriginal ending point (as indicated in the annotation anchor) by greaterthan a threshold amount (e.g., greater than twice the original distance)(act 284). If the distance is not exceeded by greater than the thresholdamount, then 50 points are added to the score (act 286), the candidateregion is extended to include the located ending point, and processingproceeds to act 288. If the distance is exceeded by greater than thethreshold amount, then processing proceeds to act 288 without adding anypoints to the score and without extending the candidate region toinclude the located ending point.

At act 288, a check is made as to whether the part of the modifiedcontent preceding the located beginning point (e.g., a set ofcharacters, such as 15, that immediately precede the located beginningpoint) matches the part of the original content preceding the originalbeginning point (as indicated in the annotation anchor). Analogous tothe discussion above, a match may exist when the features at the pointbeing analyzed are the same as or within a threshold amount of thefeatures in the annotation anchor. If the part of the modified contentpreceding the located beginning point matches the part of the originalcontent preceding the original beginning point, then 10 points are addedto the score (act 290). If the part of the modified content precedingthe located beginning point does not match the part of the originalcontent preceding the original beginning point, or if there is nolocated beginning point, then no points are added to the score.

Processing then proceeds with a check being made as to whether the partof the modified content succeeding the located ending point (e.g., a setof characters, such as 15, that immediately succeed the located endingpoint) matches the part of the original content succeeding the originalending point (as indicated in the annotation anchor) (act 292).Analogous to the discussion above, a match may exist when the featuresat the point being analyzed are the same as or within a threshold amountof the features in the annotation anchor. If the part of the modifiedcontent succeeding the located ending point matches the part of theoriginal content succeeding the original ending point, then 10 pointsare added to the score (act 294). If the part of the modified contentsucceeding the located ending point does not match the part of theoriginal content succeeding the original ending point, or if there is nolocated ending point, then no points are added to the score.

Processing then proceeds to check whether the located beginning point ispositioned in the modified content within a threshold distance of theoriginal beginning point (act 296). This threshold distance (e.g., 25%of the length of the document) is the difference in the offset of theoriginal beginning point from a particular point (e.g., the beginning ofthe document), and the offset of the located beginning point from theparticular point. If the located beginning point is positioned in themodified content within a threshold distance of the original beginningpoint, then 20 points are added to the score (act 298) and processingproceeds to act 300. However, if the located beginning point is notpositioned in the modified content within a threshold distance of theoriginal beginning point, then processing proceeds to act 300 withoutadding any points to the score. In one implementation, a number ofpoints between zero and 20 are assigned based on how far the point hasmoved (its position in the original content compared to its position inthe modified content), using a sliding scale scoring process asdiscussed in more detail below.

The addition of points in act 298 is performed to distinguish between an“acceptable” choice and a “really good” choice. For example, supposethat an annotation is attached to the word “the” in a document. The word“the” is repeated several times throughout the document, so severalcandidate regions are identified, all of which are more or less equallylikely to be the correct candidate region. By storing the distance ofthe original beginning point from the beginning of the originaldocument, this helps the process identify the correct occurrence of“the” and discount the other occurrences from being correct.

At act 300, a check is made as to whether the length of the annotationregion (from located beginning point to located ending point) haschanged by greater than a threshold amount (e.g., increased by more thantwice the original length or decreased by more than one-half theoriginal length). If the length has not changed by greater than thethreshold amount, then 50 points are added to the score (act 302);otherwise, no points are added to the score. If no beginning point islocated (in act 272) and/or no ending point is located (in act 280),then no points are added to the score. If no beginning point is locatedthen a default beginning point is determined to be the beginning of thefirst keyword identified and included in the candidate region.Similarly, if no end point is located, then a default ending point isdetermined to be the end of the last keyword identified and included inthe candidate region.

Processing then proceeds to act 304, where the score is normalized bythe maximum possible score for the keyword (which will vary based on thenumber of keywords in the annotation region). In one implementation, thescore is normalized to a scale of 0 to 100. The maximum possible score(MaxScore) for a candidate region is determined as follows:MaxScore=(keywordWeight×(no. ofkeywords−1))+(2×endPointWeight)+(2×contextWeight)+offsetWeight+lengthWeightwhere keywordWeight is the number of points added for locating a keywordthat keeps the range within the desired length (100 points in acts 270and 278), no. of keywords is the number of keywords indicated in theanchor as being in the region, endPointWeight is the number of pointsadded for locating each of the beginning point and the ending pointwithin the desired distance (50 points each in acts 278 and 286),contextWeight is the number of points added for the context of thelocated beginning point (the area preceding the beginning point) and theending point (the area succeeding the ending point) matching theoriginal context (10 points each in acts 290 and 294), offset Weight isthe number of points added for the located beginning point beingpositioned as desired (20 points in act 298), and length Weight is thenumber of points added for the length of the annotation region notchanging more than the desired amount (50 points in act 302).

In the discussion above regarding FIGS. 6 a-6 c, reference is made toparticular point values (e.g., in acts 270, 278, 286, 290, 294, 298, and302). It is to be appreciated that these specific point values areexemplary only, and that different point values could be used.

Also in the discussion above regarding FIGS. 6 a-6 c, various checks aremade and either a full number of points are added to the score or zeropoints are added to the score depending on the outcome of the check.Alternatively, rather than having either all or zero points awardedbased on the outcome of a check, a sliding scale may be implemented thatassigns different points ranging from zero up to all points based on howwell the checked value compares to the original value. For example, inchecks where distances are involved (e.g., the length of the annotationregion, or offset of the beginning point from a particular point in thecontent), the actual score (Score) may be determined as follows:Score=weight×((maxDiff−diffAmt^(1.2))÷maxDiff)where weight is the maximum number of points that may be assigned,maxDiff is the maximum possible difference, and diffAmt is the amount ofdifference between the two distances. The value diffAmt is raised topower 1.2 so that the score goes down by more as the amount of thedifference gets larger. In one implementation, if the value ofdiffAmt^(1.2) is greater than the value of maxDiff, then the value ofScore is set to zero.

Various other modifications may also be made to the process of FIGS. 6a-6 c. In one implementation, the surrounding context of an originalbeginning point and/or ending point can be used to assist in locating abeginning point and/or ending point in the modified content. Forexample, rather than relying solely on the interior of the beginningpoint in determining the location of a beginning point (in act 272), theexterior of the beginning point (the area immediately preceding thebeginning point) may be used to compare against the area immediatelypreceding the original beginning point. If the comparison results in amatch, then the beginning point may be located even if the interior ofthe beginning point is not a match. An analogous process may be used forthe ending point, except that the exterior of the ending point refers tothe area immediately succeeding the ending point. Note, however, thatbeginning and ending points located based on the surrounding context maybe given a lower weight than a beginning and ending points located basedon the interior.

Another modification that may be made is to expand anchors to aparticular default point if the beginning point or ending point cannotbe located. The particular point can vary based on the type ofannotation made (e.g., implicit or explicit), and may be, for example,the beginning or ending of a sentence, the beginning or ending of aparagraph, the beginning or ending of a section, etc. By way of example,if an annotation is anchored to a portion of a sentence in the originalcontent, but the ending point in the modified content cannot be located,then the end of the sentence that includes the last keyword (the lastkeyword that still keeps the candidate region within the desired length)may be used as the located ending point (although the number of pointsadded in act 286 may be reduced to reflect the manner in which theending point was located). By way of another example, if an annotationis anchored to a portion of a sentence in the original content, but thebeginning point in the modified content cannot be located, then thebeginning of the sentence that includes the selected keyword may be usedas the located beginning point (although the number of points added inact 278 may be reduced to reflect the manner in which the beginningpoint was located).

Another modification that may be made is to expand what words are usedas keywords. For example, proper names may be used as keywords, orcapitalized words (other than those at the beginning of a sentence) maybe used as keywords, or words in quotation marks may be used askeywords. Punctuation may also be used as a keyword. Certain punctuationmarks may be pre-determined or user-selected as being keywords (e.g.,exclamation points, question marks, quotation marks, etc.), oralternatively histograms of punctuation frequency may be generatedanalogous to those of word frequency discussed above. Additionally, the“keywords” may be limited to only letters, or may include both lettersand numbers, and optionally include other characters (for example, theampersand, section mark, etc.).

As discussed above with reference to FIG. 5, the user may be queried forinput as to whether the proposed region for anchoring an annotation iscorrect. The proposed region may be identified to the user in any of avariety of manners, such as displaying to the user the actual content inthe region, or highlighting the region in the modified content. Thisinterface may allow the user to simply indicate “yes” or “no”, oralternatively have the user manually select the region of the modifiedcontent to which the annotation should be anchored. Alternatively, theuser may be allowed to cycle through multiple different proposed regionsand select which of those regions is the correct region to which theannotation should be anchored.

Another modification that may be made is to expand on the semantics ofthe region, such as using a thesaurus to expand a particular keyword.For example, if the next keyword after the selected keyword is not foundin act 266, or would be outside the desired range in act 268, thenadditional acts may be performed to lookup synonyms for the keyword in athesaurus. These synonyms are then searched for and a determination madeas to whether they are within the desired range (analogous to acts 266and 268). If a synonym is within the desired range, then an appropriatenumber of points can be added to the score (analogous to act 270). Thenumber of points may optionally be less than the number that would havebeen added if the original keyword were found rather than a synonym.Analogous to synonyms, antonyms may also be searched for (e.g., the word“yes” has been replaced by the word “no”, or the word “beginning” hasbeen replaced by the word “ending”), although the number of points addedfor finding an antonym would optionally be less than the number addedfor finding a synonym.

Another modification may be made to attempt to identify possible splitregions. A split region refers to the region the annotation isoriginally anchored to being split into two or more parts and additionalcontent inserted between the parts. This can result in low scores forthe different parts of the annotation because the remaining parts do notsatisfy many of the distance tests that are performed (e.g., length ofthe entire region, distance from a keyword to the beginning point orending point, etc.). In this situation, the various data and scores canbe analyzed to attempt to determine such a split. This analysis toidentify a split region may always be performed, or alternatively onlysometimes (e.g., if the annotation would otherwise be orphaned). Inperforming the analysis, annotation re-anchoring engine 104 looks for“pieces” of the annotation region, such as the beginning point followedby one or more keywords, and one or more other keywords followed by theending point. If such pieces are located, engine 104 treats the entiresplit region (including the additional content inserted between the twopieces) as the annotation region.

Alternatively, each of the individual pieces may be treated as anindividual annotation region (thus splitting the original annotationinto two annotations). Intermediate points can be generated for splitregions in much the same way that beginning and ending points aregenerated, except that the intermediate points are generated within theinterior of the selected region around “significant” intermediatefeatures, such as periods and commas (which are likely to serve ascleaving points where a whole sentence or phrase may be separated fromthe preceding or succeeding sentence or phrase). These intermediatepoints are generated when the beginning and ending point features arecaptured (e.g., by annotation marking engine 102 of FIG. 1), but can beleft out of most processing by annotation re-anchoring engine 104 unlesssplitting is suspected. As intermediate points are found, they can serveas ending/beginning points for the new regions resulting from the split.

In addition, the discussion above refers to generating scores fordifferent candidate regions with the highest score being indicative ofthe closest match to the original region the annotation was anchored to(and thus the best score). Alternatively, different scoring methods maybe used so that the lowest score is the best score and indicative of theclosest match (e.g., rather than adding points in acts 270, 278, 286,290, 294, 298, and 302, points can be subtracted).

Various modifications can also be made to improve the efficiency of theprocesses described above. For example, as discussed with reference toact 142 of FIG. 3, the histogram need only be generated once perdocument. By way of another example, the indexing of keywords in themodified content (act 260 of FIG. 6 a) need only be performed once forthe content (and not repeated for each keyword in an annotation anchor).By way of another example, once a particular lo keyword is located inthe modified content for re-anchoring one annotation, it need not bere-located when re-anchoring additional annotations.

FIG. 7 illustrates an exemplary user interface 330 that may be presentedby UI module 190 of FIG. 1. The interface 330 includes an annotationdisplay window 332 in which the content of the candidate region(s) canbe displayed. Although illustrated as showing only one candidate regionat a time, alternatively multiple regions may be displayed in window 332concurrently. A region identification portion 334 identifies whichcandidate region is currently being displayed in window 332 as well asthe total number of candidate regions. Multiple user-selectable buttons336, 338, 340, and 342 are also included in interface 330. Show nextregion button 336 causes, upon selection, UI module 190 to display inwindow 332 the next candidate-region. Regions can be ordered in anymanner, such as from highest score to lowest score. Select currentregion button 338 causes, upon selection, UI module 190 to record thecandidate region currently being displayed as the region to which theannotation is to be anchored. Show region in document 340 causes, uponselection, UI module 190 to display the underlying content and drawattention to (e.g., highlight) the candidate region within theunderlying content. UI module 190 may display the underlying content inwindow 332, or alternatively open another user interface window fordisplay of the content (or alternatively take some other action, such asprint out the underlying content with the candidate region underlined).Cancel button 342 causes, upon selection, UI module 190 to close window330 and terminate the querying process for the current annotationanchor. In one implementation, if the querying process is terminatedwithout user selection of a candidate region, the annotation isorphaned.

Additional information (not shown) may also be optionally included ininterface 330. For example, the score of the region currently beingdisplayed in window 332 may also be displayed, the range of scores forthe candidate regions may be displayed, an indication of importantaspects that could not be located for the candidate region may bedisplayed (e.g., an indication that a beginning point or an ending pointcould not be found, or that a particular keyword could not be found), anindication of which keywords were found within the candidate region, andso forth.

Interface 330 is intended to be exemplary only. The components ofinterface 330 can be changed to be any of a wide variety of conventionaluser interface components. For example, rather than buttons 336-342, oneor more pull-down menus may be included that illustrate user-selectableinputs, one or more check boxes and an “ok” button may be used for theinputs, and so forth.

Additionally, UI module 190 may present an interface to the user thatlets the user adjust the region of the modified content to which theannotation is re-anchored. For example, the re-anchoring process mayidentify a region of the modified content which the user believes is toolong or too short at the beginning and/or ending. UI module 190 canpresent an interface that lets the user re-highlight the portion of themodified content to which he or she desires to have the annotationanchored. This can be done, for example, in window 332 of FIG. 7, oralternatively a separate window or dialog box. The user may be allowedto manually re-highlight the desired region, or alternatively may beallowed to input relocation commands to UI module 190 (e.g., enter anumber into a data entry field and then click a button to “extendending” by the entered number of characters or words, “shortenbeginning” by the entered number of characters or words, etc.). Any suchchanges are then stored so that the annotation is attached to thisre-highlighted portion.

FIGS. 8-12 illustrate additional exemplary user interfaces that may bepresented by UI module 190 of FIG. 1. FIG. 8 illustrates an interface350 displaying a web page 352 (e.g., written using HTML or XML) withhighlighted (shaded) annotations made on it. Highlighted region 354 is a“highlight” annotation with no additional content to it (e.g., nothingextra typed-in by the user as annotation content), while highlightedregions 356 and 358 are “note” annotations with some content. On theleft hand side an annotation display portion 360 displays an indexshowing, among other things, the keywords that were chosen from thehighlighted regions and their respective anchoring scores (all 100because they were all created on this web page). The information (e.g.,keywords and scores) 354′, 356′, and 358′ in portion 360 correspond toregions 354, 356, and 358, respectively.

FIG. 9 illustrates interface 350 displaying a page 368 that is amodified version of the original web page 352 of FIG. 8. On page 368,the highlight annotation 354 has been orphaned, and this is indicated inannotation display portion 370 (its score is less than the higher guessthreshold, but greater than the lower orphan threshold). The noteannotations 356 and 358 have both been found and repositioned correctly,however note that the region that is highlighted for note annotation 356in the modified document 368 is slightly different from the region thatwas highlighted for it in the original document 352.

FIG. 10 illustrates interface 350 displaying page 368 and is the same asFIG. 9 except that the index pane (annotation display portion 370) onthe left hand side has been scrolled to show the details for the twonote annotations 356 and 358. Note annotation 356 has aless-than-perfect score due to the region with which it is associatedbeing slightly different from the region it was originally anchored to.Note annotation 358 also has a score of less than 100 (due, for example,to changes in location of the region with which it is associated havingshifted slightly due to alterations in the content of the modifiedversion).

FIG. 11 illustrates interface 350 displaying page 368 and is similar toFIG. 9, but shows the modified web page 368 after the “guess” button 374in portion 370 has been pressed for the orphaned annotation. The “bestguess” candidate region 376 is displayed on page 368, indicating thecandidate region that the process identified as being the most likelyregion to which the orphaned annotation corresponds. The information(e.g., keywords and score) 376′ in portion 370 corresponds to region376.

FIG. 12 illustrates interface 350 displaying page 368 and is similar toFIG. 11, but shows modified web page 368 after the “best guess”candidate region has been selected as the new location for the highlightannotation and the “reattach” button 378 of FIG. 11 has been pressed.The “reattach” button 378 causes the process to calculate new anchorstatistics for the highlight annotation based on the newly selectedregion 376 and stores them with the annotation. The score for thepreviously orphaned annotation is now 100 as it is now treated as havingbeen created on web page 368.

FIG. 13 illustrates a more general exemplary computer environment 400,which can be used to implement the improved meta data managementdescribed herein. The computer environment 400 is only one example of acomputing environment and is not intended to suggest any limitation asto the scope of use or functionality of the computer and networkarchitectures. Neither should the computer environment 400 beinterpreted as having any dependency or requirement relating to any oneor combination of components illustrated in the exemplary computerenvironment 400.

Computer environment 400 includes a general-purpose computing device inthe form of a computer 402. Computer 402 can be, for example, a deviceimplementing annotation marking engine 102, annotation re-anchoringmodule 104, or content editor 154 of FIG. 1. The components of computer402 can include, but are not limited to, one or more processors orprocessing units 404, a system memory 406, and a system bus 408 thatcouples various system components including the processor 404 to thesystem memory 406.

The system bus 408 represents one or more of any of several types of busstructures, including a memory bus or memory controller, a peripheralbus, an accelerated graphics port, and a processor or local bus usingany of a variety of bus architectures. By way of example, sucharchitectures can include an Industry Standard Architecture (ISA) bus, aMicro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, aVideo Electronics Standards Association (VESA) local bus, and aPeripheral Component Interconnects (PCI) bus also known as a Mezzaninebus.

Computer 402 typically includes a variety of computer readable media.Such media can be any available media that is accessible by computer 402and includes both volatile and non-volatile media, removable andnon-removable media.

The system memory 406 includes computer readable media in the form ofvolatile memory, such as random access memory (RAM) 410, and/ornon-volatile memory, such as read only memory (ROM) 412. A basicinput/output system (BIOS) 414, containing the basic routines that helpto transfer information between elements within computer 402, such asduring start-up, is stored in ROM 412. RAM 410 typically contains dataand/or program modules that are immediately accessible to and/orpresently operated on by the processing unit 404.

Computer 402 may also include other removable/non-removable,volatile/non-volatile computer storage media. By way of example, FIG. 13illustrates a hard disk drive 416 for reading from and writing to anon-removable, non-volatile magnetic media (not shown), a magnetic diskdrive 418 for reading from and writing to a removable, non-volatilemagnetic disk 420 (e.g., a “floppy disk”), and an optical disc drive 422for reading from and/or writing to a removable, non-volatile opticaldisc 424 such as a CD-ROM, DVD-ROM, or other optical media. The harddisk drive 416, magnetic disk drive 418, and optical disc drive 422 areeach connected to the system bus 408 by one or more data mediainterfaces 426. Alternatively, the hard disk drive 416, magnetic diskdrive 418, and optical disc drive 422 can be connected to the system bus408 by one or more interfaces (not shown).

The various drives and their associated computer-readable media providenon-volatile storage of computer readable instructions, data structures,program modules, and other data for computer 402. Although the exampleillustrates a hard disk 416, a removable magnetic disk 420, and aremovable optical disc 424, it is to be appreciated that other types ofcomputer readable media which can store data that is accessible by acomputer, such as magnetic cassettes or other magnetic storage devices,flash memory cards, CD-ROM, digital versatile discs (DVD) or otheroptical storage, random access memories (RAM), read only memories (ROM),electrically erasable programmable read-only memory (EEPROM), and thelike, can also be utilized to implement the exemplary computing systemand environment.

Any number of program modules can be stored on the hard disk 416,magnetic disk 420, optical disc 424, ROM 412, and/or RAM 410, includingby way of example, an operating system 426, one or more applicationprograms 428, other program modules 430, and program data 432. Each ofsuch operating system 426, one or more application programs 428., otherprogram modules 430, and program data 432 (or some combination thereof)may implement all or part of the resident components that support thedistributed file system.

A user can enter commands and information into computer 402 via inputdevices such as a keyboard 434 and a pointing device 436 (e.g., a“mouse”). Other input devices 438 (not shown specifically) may include amicrophone, joystick, game pad, satellite dish, serial port, scanner,and/or the like. These and other input devices are connected to theprocessing unit 404 via input/output interfaces 440 that are coupled tothe system bus 408, but may be connected by other interface and busstructures, such as a parallel port, game port, or a universal serialbus (USB).

A monitor 442 or other type of display device can also be connected tothe system bus 408 via an interface, such as a video adapter 444. Inaddition to the monitor 442, other output peripheral devices can includecomponents such as speakers (not shown) and a printer 446 which can beconnected to computer 402 via the input/output interfaces 440.

Computer 402 can operate in a networked environment using logicalconnections to one or more remote computers, such as a remote computingdevice 448. By way of example, the remote computing device 448 can be apersonal computer, portable computer, a server, a router, a networkcomputer, a peer device or other common network node, and the like. Theremote computing device 448 is illustrated as a portable computer thatcan include many or all of the elements and features described hereinrelative to computer 402.

Logical connections between computer 402 and the remote computer 448 aredepicted as a local area network (LAN) 450 and a general wide areanetwork (WAN) 452. Such networking environments are commonplace inoffices, enterprise-wide computer networks, intranets, and the Internet.

When implemented in a LAN networking environment, the computer 402 isconnected to a local network 450 via a network interface or adapter 454.When implemented in a WAN networking environment, the computer 402typically includes a modem 456 or other means for establishingcommunications over the wide network 452. The modem 456, which can beinternal or external to computer 402, can be connected to the system bus408 via the input/output interfaces 440 or other appropriate mechanisms.It is to be appreciated that the illustrated network connections areexemplary and that other means of establishing communication link(s)between the computers 402 and 448 can be employed.

In a networked environment, such as that illustrated with computingenvironment 400, program modules depicted relative to the computer 402,or portions thereof, may be stored in a remote memory storage device. Byway of example, remote application programs 458 reside on a memorydevice of remote computer 448. For purposes of illustration, applicationprograms and other executable program components such as the operatingsystem are illustrated herein as discrete blocks, although it isrecognized that such programs and components reside at various times indifferent storage components of the computing device 402, and areexecuted by the data processor(s) of the computer.

Computer 402 typically includes at least some form of computer readablemedia. Computer readable media can be any available media that can beaccessed by computer 402. By way of example, and not limitation,computer readable media may comprise computer storage media andcommunication media. Computer storage media includes volatile andnonvolatile, removable and non-removable media implemented in any methodor technology for storage of information such as computer readableinstructions, data structures, program modules or other data. Computerstorage media includes, but is not limited to, RAM, ROM, EEPROM, flashmemory or other memory technology, CD-ROM, digital versatile discs (DVD)or other optical storage, magnetic cassettes, magnetic tape, magneticdisk storage or other magnetic storage devices, or any other media whichcan be used to store the desired information and which can be accessedby computer 402. Communication media typically embodies computerreadable instructions, data structures, program modules or other data ina modulated data signal such as a carrier wave or other transportmechanism and includes any information delivery media. The term“modulated data signal” means a signal that has one or more of itscharacteristics set or changed in such a manner as to encode informationin the signal. By way of example, and not limitation, communicationmedia includes wired media such as wired network or direct-wiredconnection, and wireless media such as acoustic, RF, infrared and otherwireless media. Combinations of any of the above should also be includedwithin the scope of computer readable media.

The invention has been described herein in part in the general contextof computer-executable instructions, such as program modules, executedby one or more computers or other devices. Generally, program modulesinclude routines, programs, objects, components, data structures, etc.that perform particular tasks or implement particular abstract datatypes. Typically the functionality of the program modules may becombined or distributed as desired in various embodiments.

For purposes of illustration, programs and other executable programcomponents such as the operating system are illustrated herein asdiscrete blocks, although it is recognized that such programs andcomponents reside at various times in different storage components ofthe computer, and are executed by the data processor(s) of the computer.

Alternatively, the invention may be implemented in hardware or acombination of hardware, software, and/or firmware. For example, one ormore application specific integrated circuits (ASICs) could be designedor programmed to carry out the invention.

It should be noted that the annotation anchoring described hereincaptures features of the underlying content to which the annotationcorresponds, and uses these captured features for re-anchoring theannotation to the content after it has is been modified. The anchorinformation is independent of the underlying content—no changes oradditions to the underlying content need be made in order to generatethe annotation anchor (e.g., no tags or markers need be inserted intothe underlying content to indicate where in the underlying content theannotation is to be anchored).

It should further be noted that the annotation anchoring describedherein is not tied to any particular content format. For example,various different formats exist for storing content, such as theMicrosoft Word word processing document format, the HTML format, theTagged Image File Format (TIFF), RTF, PDF, etc. The annotation anchorsare generated based on captured features from the originallanguage-level content, and, depending on the features extracted, needinvolve no format-specific structural analysis of the document, so evenif the content were to be modified and changed to a different format,the annotation could still be re-anchored to the new format.

The anchoring and re-anchoring described herein is discussed primarilywith reference to text content. However, it is to be appreciated thatthe anchoring and re-anchoring can be used with a wide variety of typesof content. With different types of content, different characteristicsof the content may be used analogously to the keywords discussed above.These different key parts of the content will vary based on the contenttype (e.g., keywords for text content, shot boundaries for videocontent, etc.).

By way of example, the underlying content 108 of FIG. 1 could be videocontent with each frame of video content being treated as analogous tocharacters in text content. The beginning point of a region of videocontent may be identified by capturing the frame number of the firstframe in the region, by capturing color histograms calculated from thefirst five frames preceding and succeeding the beginning point, etc.Similarly, the ending point of a region of video content may beidentified by capturing the frame number of the last frame in theregion, by capturing color histograms calculated from the first fiveframes preceding and succeeding the ending point, etc. For the regionbetween the beginning and ending points, various features may becaptured such as the number of frames between the two points, drasticchanges in video may be identified using conventional shot boundarydetection techniques and these shot boundaries used analogously to textkeywords, etc. An example of such a shot boundary detection technique isdiscussed in Yong Rui, Thomas S. Huang, and Sharad Mehrotra,Constructing Table-of-Content for Videos, ACM Multimedia SystemsJournal, Special Issue Multimedia Systems on Video Libraries, Vol. 7,No. 5, September 1999, pp 359-368.

By way of another example, the underlying content 108 of FIG. 1 could bea single image with each pixel in the image being treated as analogousto characters in text content. For image content, rather than a set ofsequential bits being the region, a 2-dimensional shape (e.g., arectangle, circle, triangle, etc.) may be used to define the annotationregion. Points on the 2-dimensional shape can be treated analogously tothe beginning and ending points (e.g., the top left corner of therectangle may be the beginning point and the lower right corner of therectangle may be the ending point), and actual pixel values, hashes ofpixel values, pixel offsets (e.g., from a particular point, such as anorigin), etc. may be calculated. Various features can be captured forthe region, such as a color histogram, number of pixels in the region,recognizable/trackable objects in the region, motion flow, edgefeatures, wavelet signatures, or various other standard image processingfeatures. These features can then be used to re-anchor the annotation tothe correct portion of the image (e.g., a particular individual's facein the image) despite changes in resolution of the image, cropping ofthe image, etc.

By way of yet another example, the underlying content 108 of FIG. 1could be audio content. Audio content in digital form can be separatedinto multiple frames or samples (or groups of frames and samples), andvarious features captured therefrom to be used to anchor annotations tothe audio content. In the case of analog audio content, the analog audiocan be digitized into digital form in a conventional manner (whichgenerates multiple samples). The beginning point of a region of audiocontent may be identified by capturing the frame or sample number of thefirst frame or sample in the region, by capturing the values of a numberof frames or samples preceding and succeeding the beginning point (e.g.,50 ms of audio data preceding and succeeding the beginning point), byextracting any of various features commonly used in conventional speechor audio processing programs, such as Fast Fourier Transform (FFT) togenerate frequency-domain features from the audio segment, or MelFrequency Coefficients (MFCC), etc. Similarly, the ending point of aregion of audio content may be identified by capturing the frame orsample number of the last frame or sample in the region, by capturingthe values of a number of frames or samples preceding and succeeding theending point (e.g., 50 ms of audio data preceding and succeeding theending point), by extracting any of various features commonly used inconventional speech or audio processing programs, etc. For the regionbetween the beginning and ending points, various features may becaptured such as the number of frames or samples between the two points,drastic changes in audio characteristics (e.g., changes in volume) maybe identified using conventional techniques and used analogously to textkeywords, a signature corresponding to silence/pause intervals in theregion may be generated, etc.

Additionally, the anchoring and re-anchoring described herein isdiscussed primarily with respect to explicit annotation regionidentification by a user (for example, the user highlighting orunderling a particular series of words in text content). However, theanchoring and re-anchoring is also applicable to implicit regionidentification. For example, a user may make a mark in a margin next toa paragraph and enter an annotation associated with that mark. Forimplicit region identification, various features can be captured torepresent the beginning point, ending point, and region between thepoints to which the annotation is to be anchored. Examples of suchfeatures include the closest section heading preceding or succeeding themark, the number(s) of the paragraph(s) the mark is closest to, the pagenumber of the mark, hash values of characters near the mark, and soforth. For example, the closest preceding section heading and pagenumber may be features used for the beginning point, the closestsucceeding section heading and paragraph number may be features used forthe ending point, and a hash value (one per paragraph) calculated byhashing each of the paragraph(s) closest to the mark may be usedanalogously to the keywords discussed above.

In addition, an explicit region may be automatically generated based onthe implicit proximity of the annotation to the content. For example, ifthe implicitly positioned annotation is “near” the third paragraph,choose the third paragraph as the region to which the annotation isanchored, and generate robust features from the third paragraph (as ifthe user had selected the third paragraph as the region to which theannotation corresponds).

Conclusion

Although the description above uses language that is specific tostructural features and/or methodological acts, it is to be understoodthat the invention defined in the appended claims is not limited to thespecific features or acts described. Rather, the specific features andacts are disclosed as exemplary forms of implementing the invention.

1-34. (canceled)
 35. A method of determining a portion of modified content to which an annotation corresponds, the method comprising: receiving annotation anchoring information, wherein the annotation anchoring information describes a portion of the content that the annotation corresponded to prior to the portion being modified; identifying keywords in the modified content; determining, for a region of the content that potentially corresponds to the annotation, a score for the region based at least in part on the location of the keywords in the content; repeating the determining for a plurality of identified keywords in the content; and selecting the region having the best score as the portion of content to which the annotation corresponds.
 36. A method as recited in claim 35, further comprising selecting the region having the best score as the portion of content to which the annotation corresponds if the best score exceeds a lower bound, and otherwise selecting no region as the portion of content to which the annotation corresponds.
 37. A method as recited in claim 35, further comprising selecting the region having the best score as the portion of content to which the annotation corresponds without user input if the best score exceeds a threshold value, and otherwise obtaining user input prior to selecting the region having the best score as the portion of content to which the annotation corresponds.
 38. A method as recited in claim 37, wherein the user input comprises a user request for a best guess as to the region to which the annotation corresponds.
 39. A method as recited in claim 35, further comprising selecting the region having the best score as the portion of content to which the annotation corresponds and stopping the repeating if the best score exceeds a threshold value.
 40. A method as recited in claim 35, wherein the content and the modified content are stored in two different formats.
 41. A method as recited in claim 35, wherein selecting the region having the best score comprises selecting the region having the highest score.
 42. A method as recited in claim 35, wherein determining the score for the region comprises: identifying each keyword in the annotation anchoring information; selecting one of the keywords in the annotation anchoring information; locating the selected keyword in the modified content; locating one or more additional keywords that are both identified in the annotation anchoring information and that are within a particular distance in the modified content of the selected keyword; and increasing the score for each of the one or more additional keywords.
 43. A method as recited in claim 42, wherein the particular distance comprises twice the length of the portion of the content that the annotation corresponding to prior to the portion being modified.
 44. A method as recited in claim 35, wherein determining the score for the region comprises: selecting a keyword from the annotation anchoring information; identifying, from the annotation anchoring information, one or more beginning point features for the portion of the content; locating the selected keyword in the modified content; locating a point of the modified content that precedes the selected keyword and that matches the one or more beginning point features; and determining whether to increase the score for the region including the selected keyword and the located point based at least in part on the distance between the located point and the selected keyword.
 45. A method as recited in claim 44, wherein determining whether to increase the score for the region comprises determining to increase the score for the region if the distance from the located point to the selected keyword is not greater than a threshold amount.
 46. A method as recited in claim 44, wherein the located point comprises an intermediate point.
 47. A method as recited in claim 35, wherein determining the score for the region comprises: selecting a keyword from the annotation anchoring information; identifying, from the annotation anchoring information, one or more ending point features for the portion of the content; locating the selected keyword in the modified content; locating a point of the modified content that succeeds the selected keyword and that matches the one or more ending point features; and determining whether to increase the score for the region including the selected keyword and the located point based at least in part on the distance between the selected keyword and the located point.
 48. A method as recited in claim 47, wherein determining whether to increase the score for the region comprises determining to increase the score for the region if the distance from the selected keyword to the located point is not greater than a threshold amount.
 49. A method as recited in claim 47, wherein the located point comprises an intermediate point. 50-56. (canceled)
 57. One or more computer readable media having stored thereon a plurality of instructions that, when executed by one or more processors, causes the one or more processors to perform acts comprising: receiving an annotation anchor that describes a region of original content to which an annotation corresponds; receiving a modified version of the original content; identifying one or more key parts in the annotation anchor; identifying, based at least in part on the one or more key parts, one or more regions of the modified version to which the annotation potentially corresponds; generating a score for the one or more regions; and selecting, as the region of the modified version of the original content to which the annotation corresponds, the identified region having the best score.
 58. One or more computer readable media as recited in claim 57, wherein the region of original content has been altered in the modified version.
 59. One or more computer readable media as recited in claim 57, wherein the selecting comprises: checking whether the best score exceeds a threshold value; automatically selecting the identified region having the best score if the best score exceeds the threshold value; and otherwise, querying a user for input as to whether the identified region is the region of the modified version to which the annotation corresponds.
 60. One or more computer readable media as recited in claim 57, wherein the plurality of instructions further cause the one or more processors to perform acts comprising: generating a new annotation anchor based on captured features of the identified region; and using the new annotation anchor as an anchor that describes the region of the modified version of the original content to which the annotation corresponds.
 61. One or more computer readable media as recited in claim 57, wherein the key parts are keywords, and wherein the plurality of instructions further cause the one or more processors to perform acts comprising: identifying one or more synonyms for at least one of the one or more keywords; and altering the score for a region if at least one of the one or more synonyms is included in the region. 62-74. (canceled) 